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INTRODUCTION 


The  Department  of  Defense  is  exponentially  inereasing  the  aequisition  of  joint  eomplex 
systems  that  deliver  needed  eapabilities  demanded  by  our  warfighter.  Systems  engineering  is 
the  technical  and  technical  management  process  that  focuses  explicitly  on  delivering  and 
sustaining  robust,  high-quality,  affordable  solutions.  Air  Force  leadership  has  collectively 
stated  the  need  to  mature  a  sound  systems  engineering  process  throughout  the  Air  Force. 
Gaining  an  understanding  of  the  past  and  distilling  lessons  learned  that  are  then  shared  with 
others  through  formal  education  and  practitioner  support  are  critical  to  achieving  continuous 
improvement. 

This  synopsis  conveys  the  salient  results  of  case  studies  focused  on  the  application  of 
systems  engineering  principles  within  various  programs.  Salient  results  are  conveyed  as 
learning  principles  to  facilitate  pedagogy.  But  these  results  are  also  useful  to  practicing 
engineers  and  managers  as  they  apply  systems  engineering  throughout  a  weapon  system’s 
life  cycle.  The  reader  is  encouraged  to  delve  into  the  details  contained  in  the  complete  case 
study  should  a  particular  learning  principle  relate  to  a  situation  on  your  program. 

Each  learning  principle  is  identified  as  follows: 

(short  name)  /  (learning  principle  number) 

The  short  name  key  is  as  follows: 

F-1 1 1  refers  to  the  F-1 1 1  Systems  Engineering  Case  Study 

C-5  refers  to  the  C-5A  Galaxy  Systems  Engineering  Case  Study 

GPS  refers  to  the  Global  Positioning  System  Systems  Engineering  Case  Study 

HST  refers  to  the  Flubble  Space  Telescope  Systems  Engineering  Case  Study 

TBMCS  refers  to  the  Theater  Battle  Management  Core  System  Systems  Engineering  Case 

Study 

A- 10  refers  to  the  A- 10  Thunderbolt  II  (Warthog)  Systems  Engineering  Case  Study 
GH  refers  to  the  Global  Hawk  Systems  Engineering  Case  Study 
KC-135  refers  to  the  KC-135  Simulator  Systems  Engineering  Case  Study 

The  learning  principle  title  is  highlighted  green  if  it  contains  information  related  to  an 
application  of  systems  engineering  that  one  should  consider  adopting.  The  learning  principle 
title  is  highlighted  yellow  if  it  contains  information  related  to  problems  that  should  be 
avoided  in  the  application  of  systems  engineering. 

Complete  case  studies  are  available  on  the  Air  Eorce  Center  for  Systems  Engineering  website 
at  [http://www.afit.edu/cse]. 
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CASE  STUDY  LEARNING  PRINCIPLES 


C-5/1  Requirements 

The  process  for  developing  and  documenting  the  system  performance  requirements 
integrated  the  User  (warfighter),  planners,  developers,  and  technologists  from  both  the 
government  and  industry  in  a  coordinated  set  of  trade  studies.  It  resulted  in  a  well-balanced, 
well-understood  set  of  requirements  that  fundamentally  remained  unchanged  throughout  the 
program. 

C-5/2  Total  Package  Procurement  Concept  (TPPC) 

The  Total  Package  Procurement  Concept  (TPPC)  employed  by  the  government  required  a 
fixed-price,  incentive  fee  contract  for  the  design,  development,  and  production  of  58  aircraft. 
It  included  a  clause  giving  Total  Systems  Performance  Responsibility  (TSPR)  to  the  prime 
contractor.  TPPC  was  invented  to  control  costs,  but  it  was  the  underlying  cause  of  the  cost 
overrun  and  limited  number  of  aircraft  purchased  under  the  original  contract. 

C-5/3  Weight  Empty  Guarantee 

A  Weight  Empty  Guarantee  was  included  in  the  specification  as  a  performance  requirement 
and  in  the  contract  as  a  cost  penalty  for  overweight  conditions  of  delivered  aircraft.  The 
aircraft  Weight  Empty  Guarantee  dominated  the  traditional  aircraft  performance 
requirements  (range,  payload,  etc.),  increased  costs,  and  resulted  in  a  major  shortfall  in  the 
wing  and  pylon  fatigue  life.  The  stipulation  of  a  Weight  Empty  Guarantee  as  a  performance 
requirement  had  far-reaching  and  significantly  deleterious  unintended  consequences. 

C-5/4  Independent  Review  Teams  (IRTs) 

The  Air  Eorce  C-5  Systems  Program  Office  employed  Independent  Review  Teams  (IRTs)  to 
assemble  national  experts  to  examine  the  program  and  provide  recommendations  to  the 
government.  These  problem-solving  teams  were  convened  to  gamer  the  best  advice  in 
particular  technical  areas:  stmcture  design  and  technology,  and  designs  to  achieve  useful 
service  life. 

F-111/1  Requirements  Definition  and  Management 

Ill-conceived,  difficult-to-achieve  requirements  and  attendant  specifications  made  the  E-111 
system  development  extremely  costly,  risky  and  difficult  to  manage. 

F-1 1 1/2  Systems  Architecture  and  Design  Trade-Offs 

F-1 1 1  Systems  Engineering  managers  (both  government  and  contractor)  were  not  allowed  to 
make  the  important  tradeoffs  that  needed  to  be  made  in  order  to  achieve  an  F-1 1 1  design  that 
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was  balanced  for  performance,  cost  and  mission  effectiveness  (including  survivability)  and 
the  attendant  risk  and  schedule  impacts. 

F-1 1 1/3  Communications  and  Systems  Management 

The  F-1 11  suffered  from  poor  communications  between  the  Air  Force  and  Navy  technical 
staffs,  and  from  over-management  by  the  Secretary  of  Defense  and  the  Director,  Defense 
Research  and  Engineering,  and  it  came  under  intense  congressional  scrutiny,  which  restricted 
the  System  Program  Office  (SPO)  Director  from  applying  sound  systems  engineering 
principles. 

F-1 1 1/4  Validation  and  Verffieation 

The  F-1 1 1,  like  any  complex  weapon  system  development  program  which  provides  new  war- 
fighting  capability,  had  areas  of  risk  or  deficiency  that  came  to  light  during  RDT&E  even 
though  there  was  perceived  low  risk  in  the  design.  The  E-111  development  program 
introduced  concurrency  (overlap)  between  design  validation/verification  and  production  to 
accelerate  program  schedules  because  there  was  an  urgent  need  for  the  capability.  However, 
technical  problems  uncovered  in  verification  and  validation  led  to  costly  retrofits  and 
redesign  of  the  production  versions.  The  most  notable  technical  problems  during  the  E-111 
development  were  inlet-engine  compatibility,  structural  failures  in  the  wing  carry-through 
structure,  and  the  introduction  of  the  technically  immature  digital  avionics  system  (call  the 
Mark  II)  to  replace  the  baseline  analog  avionics  system. 

F-1 1 1/5  Program  Management 

Cancellation  of  the  Navy  F-1 1  IB  in  1968,  after  bi-service  design  was  frozen  in  1964  and 
production  of  the  Air  Force  F-1 1  lA  was  well  underway,  had  a  lasting  impact  on  the  United 
States  Air  Force  (USAF)  F-1 1 1  performance  and  cost. 


GPS/1  Programs  must  strive  to  staff  key  positions  with  domain  experts. 


From  program  management  to  systems  engineering,  to  design,  to  the  manufacturing  and 
operations  teams,  the  people  on  the  program  were  well-versed  in  their  disciplines,  and  all 
possessed  a  systems  view  of  the  program.  While  communications,  working  relationships, 
and  organization  were  important,  it  was  the  ability  of  the  whole  team  at  all  levels  to 
understand  the  implications  of  their  work  on  the  system  that  was  vital.  Their  knowledge- 
based  approach  for  decision  making  had  the  effect  of  shortening  the  decision  cycle,  because 
both  the  information  was  understood  and  the  base  and  alternative  solutions  were  accurately 
presented. 

GPS/2  The  systems  integrator  must  rigorously  maintain  program  baselines. 

The  JPO  retained  the  role  of  managing  and  controlling  the  system  specification  and, 
therefore,  the  functional  baseline.  The  JPO  had  derived  and  constructed  a  mutually  agreed-to 
set  of  system  requirements  that  became  the  program  baseline  in  1973.  While  conducting  the 
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development  program,  the  GPS  team  was  able  to  make  performance/risk/cost  trade  analysis 
against  the  functional  baseline  to  control  both  risk  and  cost.  The  JPO  was  fully  cognizant  of 
the  implications  of  the  functional  requirements  on  the  allocated  baseline  because  they 
managed  the  Interface  Control  Working  Group  process.  Managing  that  process  gave  them 
firs-hand  knowledge  and  insight  into  the  risks  at  the  lowest  level. 

GPS/3  Achieving  consistent  and  continuous  high-level  support  and  advocacy  helps  funding 

stability,  which  impacts  SE  stability. 

Consistent,  continuous  high-leyel  support  proyided  requirements  and  funding  stability.  In 
this  role,  the  OSD  proyided  adyocacy  and  sourced  the  funding  at  critical  times  in  the 
program,  promoted  coordination  among  the  yarious  seryices,  and  reyiewed  and  approyed  the 
GPS  JPO  system  requirements.  OSD  played  the  central  role  in  the  establishment  and 
suryiyability  of  the  program.  The  GPS  JPO  had  clear  support  from  the  Director  of  Defense 
Deyelopment,  Research  and  Engineering  (DDR&E),  Dr.  Malcolm  Currie,  and  program 
support  from  the  Deputy  Secretary  of  Defense,  Dr.  Dayid  Packard.  Clearly  the  seryices  - 
particularly  the  Nayy  and  the  Air  Eorce  early  on,  and  later  the  Army  -  were  the  primary  users 
and  the  eyentual  customers.  Howeyer,  each  seryice  had  initial  needs  for  their  indiyidual 
programs,  or  for  the  then-current  operational  nayigation  systems.  Additionally,  the  Secretary 
of  the  Air  Eorce  proyided  programmatic  support  to  supply  manpower  and  facilities. 

GPS/4  Disciplined  and  appropriate  risk  management  must  be  applied  throughout  the 

lifecycle. 

The  GPS  program  was  structured  to  address  risk  in  seyeral  different  ways  throughout  the 
multiphase  program.  Where  key  risks  were  known  up  front,  the  contractor  and/or 
goyemment  utilized  a  classic  risk  management  approach  to  identify  and  analyze  risk,  and 
deyeloped  and  tracked  mitigation  actions.  These  design  (or  manufacturing/launch)  risks 
were  managed  by  the  office  who  owned  the  risks.  Identified  technical  risks  were  often 
tracked  by  Technical  Performance  Measures  (TPMs),  (e.g.  satellite  weight  and  Software 
Eines  of  Codes  (SEOC)),  and  addressed  at  weekly  chief  engineer’s  meetings. 

HST/1  Early  and  full  participation  by  the  customer/user  throughout  the  program  is  essential 

to  program  success. 

In  the  early  stages  of  the  HST  program  the  mechanism  was  not  well  defined  and  the  user 
community  was  initially  polarized  and  not  effectiyely  engaged  in  program  definition  and 
adyocacy.  This  ultimately  changed  for  the  better,  eyen  if  driyen  heayily  by  external  political 
and  related  national  program  initiatiyes.  Ultimately,  institutionalization  of  the  user’s  process 
for  inyolyement  ensured  powerful  representation  and  a  fundamental  stake  and  role  in  the 
program  requirements  and  requirements  management.  Oyer  time,  the  effectiyeness  of  “The 
Institute”  led  to  equally  effectiye  user  inyolyement  in  the  operational  aspects  of  system 
(deployment  and  operations)  as  well. 
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HST/2  The  use  of  Pre-Program  Trade  Studies  (“Phased  Project  Planning”  in  NASA  parlance 

at  the  time)  to  broadly  explore  technical  concepts  and  alternatives  is  essential  and  provides 

for  a  healthy  variety  of  inputs  from  a  variety  of  contractors  and  government  nSfASA)  centers. 

These  activities  cover  a  range  of  feasibility,  conceptual,  alternative  and  preliminary  design 
trades  with  cost  initially  a  minor,  then  later  a  major,  factor  as  the  process  proceeds.  For  HST, 
several  Headquarters  and  Center  organizations  funded  these  studies  and  sponsored  technical 
workshops  for  HST  concepts.  This  can  promote  healthy  or  unhealthy  competition,  especially 
when  roles  and  responsibilities  within  and  between  the  participating  management  Centers 
have  not  yet  been  decided  and  competing  external  organizations  use  these  studies  to  further 
both  technical  and  political  agendas.  Center  roles  and  missions  can  also  be  at  stake 
depending  on  political  and  or  budgetary  realities.  The  systems  engineering  challenge  at  this 
stage  is  to  “keep  it  technical,  stupid!” 

HST/3  Provision  for  a  high  degree  of  systems  integration  to  assemble,  test,  deploy  and 

operate  the  system  is  essential  to  success  and  must  be  identified  as  a  fundamental  program 

resource  need  from  early  on  (part  of  the  program  baseline). 

For  HST,  the  early  wedding  of  the  program  to  the  Shuttle,  prior  NASA  9and  of  course, 
NASA  contractors)  experience  with  similarly  complex  programs,  such  as  Apollo,  and  the 
early  requirement  for  manned,  on-orbit  servicing  made  it  hard  not  to  recognize  this  was  a  big 
SE  integration  challenge.  Nonetheless,  collaboration  between  government  engineers, 
contractor  engineers,  as  well  as  customers,  must  be  well  defined  and  exercised  early  on  to 
overcome  inevitable  integration  challenges  and  unforeseen  events. 

HST/4  Life  Cycle  Support  Planning  and  Execution  must  be  integral  from  day  one  (including 

concept  and  design  phases)  and  the  results  will  speak  for  themselves.  Programs  structured 

with  real  life  cycle  performance  as  a  design  driver  will  be  capable  of  performing  in-service 

better,  and  will  be  capable  of  dealing  with  unplanned,  unforeseen  events  (even  usage  in 

unanticipated  missions.) 

HST  likely  represents  the  benchmark  for  building-in  system  sustainment  (reliability, 
maintainability,  provision  for  technology  upgrade,  built-in  redundancy,  etc.),  all  with 
provision  for  operational  human  execution  of  functions  (planned  and  unplanned)  critical  to 
servicing  missions.  With  four  successful  service  missions  complete,  including  one  initially 
not  planned  (the  primary  mirror  repair),  the  benefits  of  design-for-sustainment,  or  life  cycle 
support,  throughout  all  phases  of  the  program,  becomes  quite  evident.  Had  this  not  been  the 
case,  it  is  not  likely  that  the  unanticipated,  unplanned  mirror  repair  could  have  even  been 
attempted,  let  alone  be  totally  successful. 

HST/5  Eor  complex  programs,  the  number  of  players  (government  and  contractor)  demands 

that  the  program  be  structured  to  cope  with  high  risk  factors  in  many  management  and 

technical  areas  simultaneously. 

Eor  HST,  there  was  heavy  reliance  on  the  contractors  (especially  Lockheed  (LMSC)  and 
Perkin-Elmer  (P-E)  and  they  each  “owned”  very  significant  and  unique  program  risk  areas. 
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In  the  critical  optical  system  area,  and  with  LM  as  the  overall  integrator,  there  was  too  much 
reliance  on  LM  to  manage  risk  in  an  area  where  P-E  was  clearly  the  technical  expert. 
Accordingly,  NASA  relied  on  LMSC  and  LMSC  relied  on  P-E  with  insufficient  checks, 
oversight  and  independence  of  the  QA  function  throughout.  While  most  other  risk  areas 
were  no  doubt  managed  effectively,  lapses  here  led  directly  to  the  primary  mirror  defect 
going  to  orbit  undetected  in  spite  of  substantial  evidence  that  could  have  been  used  to  prevent 
this  occurrence. 

TBMCS/1  Requirements  Definition  and  Management 

The  government  did  not  produce  a  Concept  of  Operations,  key  operational  performance 
parameters,  or  a  system  specification  for  the  contractor.  The  contractor  was  responsible  for 
generating  a  systems  segment  specification  that  had  performance  measures  as  goals,  but  not 
testable  requirements.  The  government  did  produce  a  technical  requirements  document  that 
defined  a  technical  approach  and  levied  certain  standards  on  the  contractor.  There  was  no 
firm  baseline  for  operational  and  system  requirements  form  which  the  system  could  be  built 
and  tested.  The  requirements  baseline  was  volatile  up  to  system  acceptance,  which  took 
place  after  TBMCS  passed  operational  test  and  evaluation. 

TBMCS/2  System  Architecture 

The  system  architecture  was  defined  at  too  high  a  level,  which  had  a  tremendous  impact  on 
system  design  and  development.  The  government’s  mandates  for  software  reuse  and  use  of 
commercial  software  products  were  often  contradictory  and  problematic  for  the  system 
development.  The  layered  system  architecture  did  support  system  evolution  and  migration  to 
modem  technologies. 

TBMCS/3  System/Subsystem  Design 

The  system  and  subsystem  design  was  seyerely  hampered  by  the  complexity  of  legacy 
applications,  misunderstanding  of  the  maturity  and  complexity  of  commercial  and  third  party 
software  products,  and  a  lack  of  understanding  of  how  the  system  would  be  employed  by  the 
user. 

TBMCS/4  System  Integration 

Systems  and  interface  integration  was  highly  complex.  The  system  integration  was  yery 
difficult  because  of  the  lack  of  detail  in  the  system  architecture  and  the  mandate  to  use 
goyemment-fumished  equipment  that  was  not  necessarily  compatible  with  commercial  off- 
the-shelf  products.  Integrating  third  party  software  products  was  an  arduous  process  and 
required  extensiye  oyersight.  The  external  system  interfaces  were  not  managed  and  were 
often  impossible  to  test  at  the  contractor’s  facility. 

TBMCS/5  Validation  and  Verification 
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The  lack  of  a  firm  requirements  baseline  made  validation  and  verification  very  difficult.  The 
program  was  schedule  driven  and  often  ran  parallel  test  processes  without  clear  measures  of 
success.  Not  being  able  to  replicate  the  operational  environment  prior  to  acceptance  test 
created  severe  problems. 


A-10/1  The  system  concept  and  preliminary  design  must  follow,  not  precede,  the  mission 

analysis. 

The  A- 10  would  have  been  a  very  different  aircraft  had  this  principle  been  violated.  The 
clear  predilection  within  the  Air  Force  at  the  time  of  needs  definition  was  for  fast  multi¬ 
purpose  aircraft.  By  concentrating  on  the  Close  Air  Support  mission,  and  focusing  on  key 
characteristics  of  that  mission,  the  early  A-X  concept  working  group  was  able  to  discover 
what  the  critical  performance  parameters  were  that  contributed  to  these  characteristics.  An 
example  of  this  approach  is  how  the  group  treated  the  key  mission  characteristic  of 
responsiveness.  While  a  contributor  towards  responsiveness  can  obviously  be  aircraft  speed, 
the  group  understood  that  responsiveness  was  even  more  dependent  on  how  close  to  the 
battlefield  the  aircraft  could  be  based  and  maintained,  how  long  the  aircraft  could  loiter 
around  the  battlefield,  and  whether  or  not  the  pilot  could  easily  communicate  and  coordinate 
with  the  ground  troops  they  were  supporting.  The  aircraft  performance  parameters  were 
analyzed  in  terms  of  alternative  design  approaches  and  aircraft  design  parameters  in  areas  of 
airframe  and  propulsion,  avionics,  armament,  and  survivability  provisions.  Once  those 
design  parameters  were  understood  and  traceable  back  to  mission  characteristics,  the  study 
group  was  able  to  evaluate  candidate  aircraft  configurations  in  terms  of  mission  and  cost 
effectiveness.  This  front  end  application  of  Systems  Engineering  led  to  well  understood 
requirements  and  provided  a  solid  foundation  with  which  to  solicit  contractor  proposals  and 
start  a  development  program. 

Part  of  the  A-X  concept  from  the  beginning  was  a  low  ownership  cost.  While  the  A- 10  did 
not  meet  its  intended  design-to-cost  goal,  the  cost  driven  approach  permeated  all  aspects  of 
the  design,  development  and  production  of  the  aircraft.  The  design  was  largely  constrained 
to  use  “existing  state-of-the-art”  technology,  avionics  were  kept  to  a  minimal  set  necessary  to 
accomplish  the  primary  missions,  and  the  design  incorporated  many  features  to  reduce  the 
maintenance  and  support  cost  for  the  aircraft.  An  example  of  this  is  the  attention  paid  to 
reducing  the  cost  associated  with  the  ammunition,  the  majority  driver  for  ownership  cost  of 
the  gun  system.  The  program  paid  more  in  the  development  effort  (due  to  multiple 
ammunition  subcontractors)  to  ensure  a  low  production  cost  would  be  attainable  when  it 
came  time  for  large  competitive  contracts  for  ammunition  to  support  operational  use. 

A- 10/2  Prototyping  can  be  used  to  help  manage  technical  and  cost  risk  at  the  system, 
■ubsystem,  and  component  level. 

There  are  mixed  feelings  throughout  the  DoD  with  respect  to  prototyping,  and  in  particular 
competitive  prototype  phases  prior  to  full  scale  development.  Most  would  agree  that 
technical  and  life  cycle  cost  risk  can  be  partially  mitigated  by  delaying  costly  commitment 
decisions  for  full  scale  development  until  after  the  basic  design  has  been  demonstrated  and 
cost  estimates  have  matured.  The  typical  downside  of  this  approach  is  that  development 
costs  rise  and  development  schedules  lengthen,  and  clearly  this  is  what  occurred  as  a  result  of 
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the  decision  to  use  the  competitive  prototype  phase  for  the  A-X  program.  Some  would  argue 
that  the  additional  development  expense  of  these  competitive  prototype  phases  makes  them 
impractical  for  large  systems  (e.g.,  bomber  aircraft,  ships).  An  example  that  can  be  taken 
away  from  the  A- 10  case  study  is  how  this  can  be  done  at  the  subsystem  and  component 
level.  The  gun  system  on  the  A- 10  was  considered  to  be  risky,  so  a  competitive  prototype 
phase  for  the  GAU-8/A  program  was  run  in  parallel  with  that  for  the  A-X  aircraft.  Further, 
the  30  mm  ammunition  was  assessed  as  having  both  technical  and  cost  risk,  so  competitive 
prototyping  was  used  at  the  subcontractor  level  within  the  GAU-8/A  program.  With  the 
expected  ownership  cost  of  the  gun  being  driven  by  the  ammunition  cost,  the  intent  was  to 
have  multiple  qualified  sources  for  ammunition  by  1978  when  a  direct  competitive  contract 
would  be  let  to  procure  ammunition  for  operational  use.  While  the  GAU-8/A  program  paid 
the  development  cost  for  this  risk  mitigation  approach,  a  good  gun  system  with  relatively  low 
operating  cost  was  obtained  as  a  result. 

A- 10/3  Clear  lines  of  responsibility  must  be  established  to  ensure  successful  integFation, 
especially  when  multiple  pro^ims  are  involved. 

The  Air  Force  and  the  program  office  understood  what  the  A- 10  system  integration  risks 
were,  and  took  steps  to  mitigate  them.  These  risks  were  associated  with  the  integration  of  a 
large  caliber  internal  gun  system,  and  the  ammunition  associated  with  the  gun.  The  aircraft, 
the  gun,  and  the  ammunition  were  all  parallel  development  efforts,  and  it  was  important  that 
they  all  came  together  if  the  A-10  was  to  provide  the  capability  the  Air  Force  needed.  The 
Air  Force  addressed  this  integration  risk  directly  by  how  they  set  up  responsibility  for  the 
various  programs.  Managerial  responsibility  for  the  GAU-8/A  gun  system  was  given  to  the 
A-10  program  office,  and  the  memorandum  of  understanding  establishing  this  relation 
between  the  aircraft  and  gun  program  was  later  expanded  to  include  the  ammunition  as  well. 
The  ammunition  suppliers  were  established  under  a  subcontract  relation  to  GE,  the  prime 
contractor  for  the  gun  system.  This  made  GE  responsible  for  the  total  gun  system  and  a 
single  program  office  responsible  for  the  combined  aircraft  and  gun  system.  While  the  Air 
Eorce  did  not  establish  this  same  type  of  relationship  with  the  engine  program,  the  TE-34 
engine  was  already  under  development  for  another  aircraft  at  the  time  the  A-X  program 
began.  Eurther,  the  engine  placement  off  the  fuselage  of  the  A-10  allowed  for  relatively  easy 
integration  with  the  airframe. 


A- 10/4  The  government  must  ensure  the  contractor  is  able  to  “Walk  the  Talk”  when  it  comes 
to  production. 

In  reviewing  the  abilities  of  the  contractor  to  execute  the  contract,  the  Air  Eorce  failed  to 
identify  a  number  of  issues  that  might  well  have  doomed  the  program  to  failure.  Both  before 
and  after  the  awarding  of  the  contract  the  company  was  in  trouble.  The  government’s  pre¬ 
award  survey  of  Eairchild  Hiller  examined  their  capacity,  capability  and  financial  condition, 
but  failed  to  recognize  some  of  the  risk  elements  and  concerns  that  would  be  noted  some  3 
years  later.  Eairchild  had  failed  to  adequately  invest  in  equipment  and  its  workforce,  and  its 
management  and  organization  had  deficiencies.  The  Air  Eorce  made  a  number  of 
recommendations  that  were  followed,  and  the  company  was  able  to  produce  the  aircraft; 
however,  it  ultimately  put  the  company  in  a  position  from  which  it  could  not  recover. 
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A- 10/5  Successful  design,  development  and  produetion  is  not  enough  to  sustain  a  system 
throughout  its  life  cyele. 

By  most  aeeounts,  the  A- 10  was  a  well  designed  and  well  built  aireraft.  Its  performanee  in 
Desert  Storm  eertainly  supports  this  elaim.  Prior  to  Desert  Storm,  however,  the  Air  Foree 
had  eommitted  to  retiring  the  A- 10  and  this  adversely  affeeted  the  sustainment  efforts  for  the 
aireraft.  While  the  Air  Foree  reversed  the  deeision  to  retire  the  A- 10  shortly  after  Desert 
Storm,  it  did  not  make  the  investments  in  maintenanee  and  modifieations  that  should  have 
followed  that  reversal.  Multiple  sourees  noted  systemie  negleet  of  the  inspeetion, 
maintenanee,  and  attention  to  serviee  life  issues  throughout  the  late  1980’s  and  most  of  the 
1990’s.  Required  inspeetions  were  not  eondueted,  early  fatigue  indieations  at  Wing  Station 
23  and  other  loeations  were  effeetively  ignored,  and  eritieal  data  regarding  aetual  serviee  life 
was  no  longer  being  obtained.  This  was  further  impaeted  by  major  disruptions  to  the 
sustainment  program  offiee  as  a  result  of  the  1995  Base  Realignment  and  Closure  deeisions. 
The  program  offiee  reported  80%  loss  of  experieneed  personnel  due  to  the  work  transfer 
from  MeClellan  to  Flill  AFB,  and  repeated  turnover  at  the  System  Program  Direetor  and 
Chief  Engineer  level  was  equally  problematie.  The  Air  Foree  lost  awareness  of  the  struetural 
health  of  the  A- 10  fleet,  and  the  intended  repair  for  the  aireraft,  whieh  did  not  address  the  full 
seope  of  issues  required  to  provide  the  required  life  extension,  proved  to  be  eostly  and 
ineffeetive  in  providing  the  required  serviee  life.  The  end  result  was  a  deeision  to 
manufaeture  new  wings  for  all  thin  skin  aircraft  remaining  in  the  fleet.  Had  the  Air  Foree 
maintained  awareness  of  the  struetural  health  of  the  A- 10,  the  deeision  to  re-wing  eould  have 
been  made  earlier,  possibly  obviating  the  need  for  the  wing  eomponents  of  the  HOG  UP 
program.  This  likely  would  have  resulted  in  an  overall  eost  savings  and  earlier  operational 
eapability  for  the  re-winged  aireraft. 

A-10/6  “If  the  polities  don’t  fly,  the  sfstem  never  will.”* 

The  quote  defining  this  Learning  Prineiple  is  one  of  the  heuristies  from  the  text  The  Art  of 
Systems  Architecting  by  Mark  Maier  and  Eberhart  Reehtin.  In  that  text  the  authors  devote  an 
entire  ehapter  to  “The  Politieal  Proeess  and  Systems  Arehiteeting”.  Other  heuristies  from 
that  ehapter  inelude; 

•  Affordability  is  deeided  by  whiehever  side  has  the  most  votes; 

•  A  strong,  eoherent  eonstitueney  is  essential; 

•  Teehnieal  problems  beeome  political  problems; 

•  The  best  engineering  solutions  are  not  neeessarily  the  best  politieal  solutions. 

All  of  these  heuristies  ean  be  applied  in  retrospeet  to  the  A- 10  as  it  was  beset  with  politieal 
fighting  for  the  entire  life  of  the  program.  Early  on,  internal  Air  Eoree  polities  made  it 
diffieult  to  pursue  a  speeialized  CAS  aireraft  when  the  majority  of  the  serviee  favored  fast 
multi-purpose  fighters.  Later,  it  faeed  diffieulties  justifying  its  existenee  alongside  the  Army 
attaek  helieopters;  however,  it  may  have  been  the  very  existenee  of  these  advaneed  helieopter 
development  programs  that  allowed  it  to  overeome  resistanee  to  the  CAS  aireraft  within  the 
Air  Eoree.  Still  later,  the  A- 10  was  foreed  to  walk  a  tight  line  between  those  that  wanted  a 
simpler,  lower  eost  aireraft,  and  those  that  didn’t  want  the  A- 10  beeause  of  its  shorteomings 


*  Maier,  Mark  &  Reehtin,  Eberhardt,  The  Art  of  Systems  Architecting,  2"‘‘  ed.,  CRC  Press,  New  York,  2002. 
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at  night  and  in  adverse  weather  (brought  about  by  a  “lean”  avionics  package  to  keep  costs 
down).  Politics  shaped  the  A- 10  program  structure,  most  notably  in  the  adoption  of  a 
politically  attractive  competitive  prototyping  phase  associated  with  both  the  A-X  aircraft  and 
the  GAU-8/A  gun.  Congressional  politics  forced  an  additional  competition  with  the  A-7D 
even  after  the  competitive  fly-off  between  the  YA-9  and  YA-10,  and  attempted  to  force 
another  show  down  with  the  Piper  Enforcer.  Finally,  congressional  politics  attempted  to 
derail  redistribution  of  production  required  to  address  deficiencies  associated  with  the 
Farmingdale  NY  plant.  The  proponents  of  the  A- 10  won  some  of  these  battles,  and  lost  some 
as  well,  but  persevered  in  cobbling  together  the  “strong,  coherent  constituency”  considered 
essential  to  the  success  of  the  program. 

GH/1  While  an  “evolving  requirements”  strategy  may  be  an  excellent  choice  for  a  concept 
demonstration  program,  it  is  an  unwise  strategy  for  an  EMD  program. 

Typically,  a  System  Specification  derived  from  the  SRD  is  placed  on  contract  at  the 
beginning  of  EMD  to  establish  the  contractual  basis  for  the  subsequent  design  and 
verification  activities  and  support  the  definition  of  the  scope  and  cost  of  the  development 
program.  With  the  spiral  development  approach  that  was  employed,  there  was  no  contractual 
System  Specification  at  the  start  of  Spiral  1  or  Spiral  2  (each  of  which  constituted  significant 
EMD  programs),  just  a  draft  SRD  that  contained  top-level  requirements.  The  intent  was  to 
evolve  to  the  baseline  technical  requirements  at  the  completion  of  EMD,  thus  providing  the 
contractor  with  greater  contractual  latitude,  similar  to  the  ACTD  program.  However,  at  this 
point  in  the  program,  the  user  had  specific  performance  requirements  that  needed  to  be  met, 
and  these  were  not  necessarily  consistent  with  available  funding.  The  result  was  a  choice  of 
“the  devil’s  alternative”:  either  breach  the  program  cost  ceiling  or  fail  to  meet  the  user 
requirements.  Not  having  a  firm  set  of  technical  requirements  also  increased  the  program  risk 
relative  to  requirements  creep  and  baseline  verification  of  engineering  content.  As  a  result, 
the  program  experienced  a  Nunn-McCurdy  cost  breach  in  excess  of  the  25  percent  threshold 
during  Spiral  2  (Block  20)  of  EMD. 

GH/2  An  overall  systems  engineering  strategy  must  be  defined  as  early  as  possible  in  i| 
program  and  must  be  consistent  with  the  acquisition  ^proach  and  progum  needs. 

During  the  ACTD  phase,  the  contractor  had  a  small  systems  engineering  organization 
consisting  of  systems  engineers  located  within  each  of  the  Integrated  Product  Teams  (IPTs). 
The  contractor  integrated  the  systems  engineering  function  partly  into  the  technical 
management  structure  wherein  some  of  the  IPT  leads  were  experienced  in  the  systems 
engineering  discipline.  The  Chief  Engineer  created  this  organization  and  chose  the  IPT  leads 
based  on  leadership  capability,  interpersonal  skills,  and  technical  excellence  in  their  area  of 
expertise  (systems  engineering,  design,  analysis,  manufacturing/assembly,  logistics,  cost, 
ground  segment,  communications,  payloads,  etc.)  In  this  concept,  the  Chief  Engineer  was  the 
individual  with  primary  responsibility  to  define  and  execute  the  systems  engineering  process. 
At  the  start  of  EMD,  the  systems  engineering  function  was  largely  integrated  into  the  IPTs. 
About  two  years  into  EMD,  the  contractor  reorganized,  and  the  systems  engineers  were 
centralized  into  a  Systems  Engineering  Integration  Team  (SEIT).  The  SEIT  was  viewed  by 
the  IPTs  as  a  “non-value  added”  activity  and  “watch  dog”  that  imposed  workload  that 


Approved  for  Public  Release;  Distribution  Unlimited 


detracted  from  the  IPX’s  ability  to  accomplish  required  tasks.  Consequently,  the  SEIT  had 
very  limited  ability  to  successfully  impact  the  conduct  of  the  program.  This  is  supported  by 
the  point  that,  when  funding  problems  drove  program  cutbacks,  systems  engineering  proved 
to  be  an  easy  target.  As  a  result,  the  Spiral  2  effort  was  lacking  in  processes  necessary  to 
support  the  efficient  completion  of  an  EMD  program. 


GH/3  The  more  an  acquisition  strategy  plans  concurrency  between  developing  the  validate 
baseline  and  production,  the  hi;^er  the  risk  of  future  cost  increases  and  schedule  delays. 

Before  a  program  proceeds  into  production,  there  is  a  need  that  all  participants  understand 
how  the  product  baseline,  expressed  in  terms  of  performance  capabilities  and  limitations, 
meets  the  functional  baseline,  expressed  in  terms  of  technical  requirements.  If  this  mutual 
understanding  does  not  exist,  then  there  is  a  significant  risk  that  the  user  needs  will  not  be 
met. 


pH/4  The  more  complex  the  program,  the  more  critical  it  is  that  an  IMS  links  all  aspects  o|' 
pie  prognm,  including  the  lower-tiered  schedules. 

The  Spiral  1  EMD  Statement  of  Work  (SOW)  required  that  the  contractor  create  and 
maintain  an  IMS  that  showed  interdependencies  and  critical  paths.  The  contractor  responded 
by  creating  a  multitude  of  IMSs:  one  for  each  Spiral  and  one  for  each  second  tier  IPX.  The 
IMSs  were  independent  of  each  other,  did  not  address  subcontracted  efforts,  and  there  was  no 
single  overarching  IMS  that  integrated  all  the  lower-tiered  IMSs.  The  Global  Hawk  program 
was  complex  with  multiple  spirals  occurring  simultaneously,  and  few  individuals,  if  any, 
knew  how  all  the  “pieces”  fit  together.  The  lack  of  a  single,  integrated  schedule  showing 
overall  program  critical  paths  and  interdependencies  made  it  impossible  for  program 
management  to  truly  assess  the  overall  program  status.  In  reality,  no  one  knew  how  one 
schedule  slip  impacted  another  and  whether  someone  else’s  critical  path  was  being  violated. 


GH/5  Contractual  programmatic  and  technical  requirements  must  be  clearly  defined, 
understood,  and  executed  by  all  parties  to  include  the  areas  of  airworthiness  certification  an4 
■oftware  development  and  verification. 

In  order  to  avoid  disputes  between  the  contractor  and  Air  Force  customer,  it  is  necessary  to 
contractually  define  key  programmatic  and  technical  requirements.  The  contractual  wording 
needs  to  be  clear  and  unambiguous.  Also,  particular  care  needs  to  be  taken  to  ensure  that  all 
parties  share  the  same  understanding  of  the  requirements  and  that  the  requirements  are 
executed  in  a  timely  fashion.  This  principle  was  violated  in  two  key  areas:  airworthiness 
certification  and  software  development/validation. 

GH/6  Development  testing  must  follow  a  disciplined  engineering  process  or  cost,  schedule, 
and  technical  risks  may  be  incurred. 

The  OSD  assessment  in  support  of  the  Nunn-McCurdy  recertification  concluded  that  the 
program  development,  test,  and  evaluation  (DT&E)  strategy  was  insufficient  to  reduce 
program  risk  and  support  knowledge-based  decisions.  Lower-level  development  tests  were 
typically  not  used  as  a  building  block  to  system  test,  entry  and  exit  criteria  were  typically 
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missing,  test  planning  and  test  procedures  were  inadequate,  critical  test  parameters  were 
often  not  verified,  test  assets  were  lacking,  and  systems  were  not  assessed  against  their 
intended  environment.  In  short,  OSD  concluded  that  the  lack  of  a  disciplined  engineering 
approach  to  development  testing  was  a  contributor  to  the  Nunn-McCurdy  breach. 


KC  -135/1  Systems  Engineering  must  translate  program  goals  and  objectives  into  clearly 
defined  and  verifiable  system  requirements,  focusing  on  the  entire  life  cycle. 

Key  elements  of  requirements  allocation  include  early  involvement  by  the  support  contractor 
with  the  aircraft  systems  prime  contractor(s)  to  ensure  simulator  specific  requirements  are 
addressed.  Because  aircraft  upgrades  are  identified  by  the  KC-135  Program  Office  at  Tinker 
AFB  there  are  roadmap  meetings  held  at  Tinker  where  upcoming  modifications  to  the  KC- 
135  aircraft  are  discussed.  The  KC-135  ATS  O&M  contractor  and  the  Training  System 
Program  Office  at  Ogden  now  are  present  to  assess  those  modifications  and  ensure  the  ATS 
requirements  are  included  in  the  early  planning  process. 


KC- 135/2  The  Systems  Engineering  process  must  be  structured  to  properly  mitigate 
challenges  generated  by  third-party  Modification  Contractors. 

The  KC-135  O&M  contract  states  the  prime  contractor,  FlightSafety,  is  responsible  for 
meeting  the  overall  performance  requirements  of  the  training  system  including  trained 
students  that  meet  Government  standards.  Competitive  contracting  for  early  OFT  upgrades 
did  not  formally  involve  buy-in  from  the  O&M  contractor  who  retains  ultimate  responsibility 
for  providing  a  “guaranteed”  student  to  the  Government.  This  was  recognized  and  remedied 
in  future  contracts  by  involving  the  support  contractor  in  a  more  disciplined  manner  to  ensure 
early  involvement  in  the  development  effort.  For  example,  the  requirement  for  a  Performance 
Work  Statement  (PWS)  has  evolved  that  identifies  specific  tasks  to  be  accomplished  by  the 
KC-135  ATS  prime  in  order  to  ensure  training  system  requirements  are  identified  at  the  ATS 
level  and  properly  allocated  to  the  various  subsystems  under  development. 


KC  -135/3  Systems  engineers  must  be  responsible  for  ensuring  that  all  stakeholders  arq 
involved  during  key  decision  technical  planning  and  execution  process  reviews. 

The  KC-135  program  requires  a  tailored  set  of  formal  reviews  be  held  during  the 
development  phase  that  is  based  on  the  size  and  complexity  of  the  modification  program. 
These  reviews  ensure  that  the  entire  KC-135  ATS  team  is  working  to  the  same  requirements, 
designing  and  developing  the  correct  modifications,  adequately  testing  the  modifications  and 
generating  the  appropriate  courseware  changes.  The  reviews  employed  throughout  the 
modification  development  and  verification  efforts  may  include  a  Systems  Requirements 
Review  (SRR),  Preliminary  Design  Review  (PDR),  Critical  Design  Review  (CDR),  Test 
Readiness  Review  (TRR),  Required  Assets  Available  Review  (RAAR)  and  In  Process 
Reviews  (IPR).  KC-135  simulator  reviews  are  structured  to  ensure  that  the  KC-135 
Simulator  team  has  mutual  expectations  and  understanding  of  requirements  and  that  the 
contractor’s  proposed  preliminary  designs  and  program  plan  satisfies  the  development 
specification. 
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KC- 135/4  Integrated  logistics/maintainability  si^ort  structure  avoids  parts  obsolescence  anj 
diminishing  manufacturer  supply  issues. 

As  with  aircraft  systems,  training  systems  must  also  have  detailed  technical  plans  that 
integrate  a  logistics/maintainability  support  structure  to  ensure  continued  future  operation. 
During  each  technical  planning  activity,  systems  engineering  must  be  concerned  with  the 
total  support  of  the  system  to  assure  its  economic  and  effective  operation  throughout  its  life 
cycle.  Logistics  objectives  for  the  program  need  to  be  included  in  these  technical  plans  to 
ensure  achieving  stated  readiness  objectives  such  as  system  availability,  programmed  flying 
training  throughput,  establishment  of  Reliability  and  Maintainability  performance 
requirements  needed  to  support  readiness  objectives,  and  emphasizing  logistics  support 
considerations  in  all  design  trade  studies. 


KC- 135/5  Simulator  modeling  data/modification  requires  verification  and  validation  to 
^sure  aircraft-like  flying  qualities. 

AMC  and  FlightSafety  personnel  have  also  developed  a  test  and  evaluation  process  that 
promotes  confidence  that  KC-135  simulator  modifications  “fly”  like  the  KC-135  aircraft. 
Even  though  the  systems  integrating  contractor  and  the  Government  quantitatively  specify 
many  requirements,  the  final  evaluation  of  the  training  realism  is  still  subjectively  validated. 
In  the  end,  the  trainer  model  must  be  correct  enough  to  allow  training,  which  means  it’s  a 
judgment  call  by  the  test  crews  and  Air  Force  instructors  about  the  system’s  “training  value.” 
The  KC-135  ATS  team  has  implemented  an  approach,  which  utilizes  no  more  than  one  or 
two  contractor  instructor  (FlightSafety)  pilots  and  one  Air  Force  instructor  pilot  to  minimize 
extended  test  periods  and  facilitate  reaching  consensus  on  a  modification’s  training  value. 
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